Psychological safety and perceived organizational support in emergency medicine residencies.

Background: The clinical learning environment (CLE) is a key focus of the Accreditation Council of Graduate Medical Education. It impacts knowledge acquisition and professional development. A previous single-center study evaluated the psychological safety and perceived organizational support of the CLE across different specialties. Building on this work, we explored and evaluated psychological safety and perceived organizational support across multiple heterogeneous emergency medicine (EM) residencies to identify trends and factors affecting perceptions of the CLE. Methods: Using the Psychological Safety Scale (PSS) and Survey of Perceived Organizational Support (SPOS), residents from seven U.S. EM residencies were surveyed using REDCap software from September through November 2021, with 300 potential respondents. As an adjunct to these surveys, three open-ended questions were included regarding features of their learning environments. Results: A total of 137 out of 300 residents completed the survey. The overall response rate was 45.7%. There was a variable response rate across programs (26.0%-96.7%). Pooled results demonstrate an overall positive perception of CLEs, based on positive mean responses (i.e., "Members of my department are able to bring up problems and tough issue" had a mean of 4.2 on a 5-point Likert scale). Open responses identified teaching, collegiality, and support from program leadership as supportive features of the CLE. Confrontational interdisciplinary communication, a sense of being devalued, and off-service rotations were identified as threats or areas for improvement to the CLE. Conclusions: PSS and SPOS scores were generally positive in this multi-institution study, consistent with the prior single-institution study indicating that EM is often considered psychologically safe and supportive. EM training programs can consider using the PSS/SPOS to audit their own programs to identify areas for improvement and foster supportive features already in place.

Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study.

Introduction: Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review's value to a journal editor's decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice (Special Issue) developed and studied the holistic editor's scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration. Methods: Using peer-review guidelines from several journals, the Special Issue's editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane's validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making. Results: Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor's rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor's formal decision. Conclusion: Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.

A Collaborative Approach to Mentored Peer Reviews Sponsored by the Council of Residency Directors in Emergency Medicine.

Introduction: Historically, there have been no systematic programs for teaching peer review, leaving trainees to learn by trial and error. Recently, a number of publications have advocated for programs where experienced reviewers mentor trainees to more efficiently acquire this knowledge. Objective: Our goal was to develop an introductory learning experience that intentionally fosters peer-review skills. Methods: The Council of Residency Directors in Emergency Medicine (CORD) offered education fellowship directors the opportunity to mentor their fellows by reviewing submitted manuscript(s) supplemented by educational material provided by their journal. Reviews were collaboratively created. The decision letter that was sent to manuscript authors was also sent to the mentees; it included all reviewers' and editor's comments, as feedback. In 2022, fellows received a post-experience survey regarding prior experiences and their perspectives of the mentored peer-review experience. Results: From 2020-2022, participation grew from 14 to 30 education fellowships, providing 76 manuscript peer reviews. The 2022 survey-response rate of 87% (20/23) revealed that fellows were inexperienced in education scholarship prior to participation: 30% had authored an education paper, and 10% had performed peer review of an education manuscript. Overall, participants were enthusiastic about the program and anxious to participate the following year. In addition, participants identified a number of benefits of the mentored experience including improved understanding of the scholarship process; informing fellows' scholarly pursuits; improved conceptualization of concepts learned elsewhere in training; and learning through exposure to scholarship. Conclusion: This program's early findings suggest that collaboration between academic societies and interested graduate medical education faculty has the potential to formalize the process of learning peer review, benefitting all involved stakeholders.

The Impact of a Medical Education Research Faculty Development Program on Career Development, Through the Lens of Social Cognitive Career Theory.

OBJECTIVES: The Medical Education Research Certificate at the Council of Residency Directors in Emergency Medicine (MERC at CORD), a specialized adaptation of the Association of American Medical Colleges MERC program, provides faculty development in education research in emergency medicine. However, its long-term influence on career development remains unknown. Our study explored the impact of MERC at CORD on career development through the lens of social cognitive career (SCC) theory. METHODS: This was a prospective qualitative study using a constructivist/interpretivist paradigm to assess long-term career development outcomes. A purposeful randomized stratified sampling strategy of MERC at CORD graduates (2011-2014) ensured diversity of representation (sex, region, number of research publications, and project group leadership). Subjects were invited by e-mail to participate in semistructured phone interviews. Thematic analysis by two independent reviewers followed an iterative process until saturation was reached. RESULTS: Twelve graduates were interviewed. All engaged with MERC at CORD early in their careers with minimal previous education research experience. Currently, all hold medical education leadership positions. Graduates had a mean of 19.3 publications (range = 9-43). Themes explaining reasons for participating in MERC at CORD include: desire for education research skills, recommendation of mentors/colleagues, and accessibility. Themes citing the program's value to career development include networking/collaboration, mentorship, informational framework to build upon, and the application of theoretical knowledge through experiential learning. MERC at CORD impacted career development aligning with the core domains of SCC theory including self-efficacy, outcome expectations, and goals. CONCLUSION: MERC at CORD enhanced the long-term career development of participants by providing a core knowledge framework in a mentored, experiential learning environment. Participants identified themes aligned with SCC theory as influential in their long-term career advancement in medical education including the development of education research skills, successful completion of education research, career acceleration, promotion, niche development, and formulation of professional goals.

A Social Network Analysis of the Western Journal of Emergency Medicine Special Issue in Educational Research and Practice.

INTRODUCTION: Scholarship and academic networking are essential for promotion and productivity. To develop education scholarship, the Council of Emergency Medicine Directors (CORD) and Clerkship Directors of Emergency Medicine (CDEM) created an annual Special Issue in Educational Research and Practice of the Western Journal of Emergency Medicine. The objective of this study was to evaluate the network created by the special Issue, and explore changes within the network over time. METHODS: Researchers used bibliometric data from Web of Science to create a social network analysis of institutions publishing in the first four years of the special issue using UCINET software. We analyzed whole-network and node-level metrics to describe variations and changes within the network. RESULTS: One hundred and three (56%) Accreditation Council for Graduate Medical Education-accredited emergency medicine programs were involved in 136 articles. The majority of institutions published in one or two issues. Nearly 25% published in three or four issues. The network analysis demonstrated that the mean number of connections per institution increased over the four years (mean of 5.34; standard deviation [SD] 1.27). Mean degree centralization was low at 0.28 (SD 0.05). Network density was low (mean of 0.09; SD 0.01) with little change across four issues. Five institutions scored consistently high in betweenness centrality, demonstrating a role as connectors between institutions within the network and the potential to connect new members to the network. CONCLUSION: Network-wide metrics describe a consistently low-density network with decreasing degree centralization over four years. A small number of institutions within the network were persistently key players in the network. These data indicate that, aside from core institutions that publish together, the network is not widely connected. There is evidence that new institutions are coming into the network, but they are not necessarily connected to the core publishing groups. There may be opportunities to intentionally increase connections across the network and create new connections between traditionally high-performing institutions and newer members of the network. Through informal discussions with authors from high-performing institutions, there are specific behaviors that departments may use to promote education scholarship and forge these new connections.

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail.

INTRODUCTION: Interest is growing in specialty-specific assessments of student candidates based on clinical clerkship performance to assist in the selection process for postgraduate training. The most established and extensively used is the emergency medicine (EM) Standardized Letter of Evaluation (SLOE), serving as a substitute for the letter of recommendation. Typically developed by a program's leadership, the group SLOE strives to provide a unified institutional perspective on performance. The group SLOE lacks guidelines to direct its development raising questions regarding the assessments, processes, and standardization programs employ. This study surveys EM programs to gather validity evidence regarding the inputs and processes involved in developing group SLOEs. METHODS: A structured telephone interview was administered to assess the input data and processes employed by United States EM programs when generating group SLOEs. RESULTS: With 156/178 (87.6%) of Accreditation Council of Graduate Medical Education-approved programs responding, 146 (93.6%) reported developing group SLOEs. Issues identified in development include the following: (1) 84.9% (124/146) of programs limit the consensus process by not employing rigorous methodology; (2) several stakeholder groups (nurses, patients) do not participate in candidate assessment placing final decisions at risk for construct under-representation; and (3) clinical shift assessments don't reflect the task-specific expertise of each stakeholder group nor has the validity of each been assessed. CONCLUSION: Success of the group SLOE in its role as a summative workplace-based assessment is dependent upon valid input data and appropriate processes. This study of current program practices provides specific recommendations that would strengthen the validity arguments for the group SLOE.

Effect of Interpolated Questions on Podcast Knowledge Acquisition and Retention: A Double-Blind, Multicenter, Randomized Controlled Trial.

STUDY OBJECTIVE: We determine whether interpolated questions in a podcast improve knowledge acquisition and retention. METHODS: This double-blinded controlled trial randomized trainees from 6 emergency medicine programs to listen to 1 of 2 versions of a podcast, produced de novo on the history of hypertension. The versions were identical except that 1 included 5 interpolated questions to highlight educational points (intervention). There were 2 postpodcast assessments, 1 within 48 hours of listening (immediate posttest) and a second 2 to 3 weeks later (retention test), consisting of the same 15 multiple-choice questions, 5 directly related to the interpolated questions and 10 unrelated. The primary outcome was the difference in intergroup mean assessment scores at each point. A sample size was calculated a priori to identify a significant interarm difference. RESULTS: One hundred thirty-seven emergency medicine trainees were randomized to listen to a podcast, 69 without interpolated questions (control) and 68 with them (intervention). Although no significant difference was detected at the immediate posttest, trainees randomized to the interpolated arm had a significantly higher mean score on the retention test, with a 5.6% absolute difference (95% confidence interval 0.2% to 10.9%). For the material covered by the interpolated questions, the intervention arm had significantly higher mean scores on both the immediate posttest and retention tests, with absolute differences of 8.7% (95% confidence interval 1.6% to 15.8%) and 10.1% (95% confidence interval 2.8% to 17.4%), respectively. There was no significant intergroup difference for noninterpolated questions. CONCLUSION: Podcasts containing interpolated questions improved knowledge retention at 2 to 3 weeks, driven by the retention of material covered by the interpolated questions. Although the gains may be modest, educational podcasters may improve knowledge retention by using interpolating questions.

Emergency Medicine Standardized Letter of Evaluation (SLOE): Findings From the New Electronic SLOE Format.

BACKGROUND: Emergency medicine (EM) uses a standardized template for residency application letters of recommendations. In 1997, the Standardized Letter of Recommendation was developed with categories for applicant comparison. Now named the Standardized Letter of Evaluation (SLOE), it is a universal requirement in the EM residency application process. In 2016, a website called "eSLOE" was launched for credentialed academic EM faculty to enter applicant data, which generates a SLOE. OBJECTIVE: This article outlines website creation for the eSLOE and its successful national implementation in the 2016-2017 EM residency application cycle. We analyzed current trends in applicant assessments from the eSLOE data and compared them to prior data. METHODS: Data from 2016-2017 were sorted and analyzed for each question on the eSLOE. An analysis of Global Assessment and Qualifications for EM rankings, clerkship grade, and comparison with prior SLOE data was performed. RESULTS: Analysis of 6715 eSLOEs for 3138 unique applicants revealed the following Global Assessment rankings: top 10%, top one-third, middle one-third, and lower one-third. There was less spread with the distribution for clerkship grade and Qualifications for EM. The 2011-2012 standard letter of recommendation global assessment data, with top 10%, top one-third, middle one-third, and lower one-third, also revealed top-clustered results with less spread compared with the ranking usage in 2016-2017. CONCLUSIONS: Results indicate an improved spread of all rank categories for Global Assessment, enhancing the eSLOE's applicant discrimination. There has been an overall improvement in rank designation when compared with previously published data.

Time to Reevaluate the Away Rotation: Improving Return on Investment for Students and Schools.

Away rotations are common among senior medical students preparing to apply for residency. For competitive specialties, multiple away rotations may be viewed as a de facto requirement for a strong application. Although away rotations are often valuable learning experiences, a variety of noneducational factors motivate students to enroll, including the need for letters of recommendation, students' uncertainty regarding their strength as applicants, perceived competitiveness of residency applications, and conflicting guidance from advisors.Students who enroll in away rotations often benefit from a novel educational environment, opportunities for career exploration, and the chance to assess "fit" with a program. Yet away rotations also come at a significant cost. Students must deal with the time and expenses of rotating at multiple institutions. The application process for away rotations is expensive, disjointed, and inefficient. Students must work hard to make an impression on host institution faculty, and risk hurting their residency applications with a negative letter of recommendation.To reduce the burden of away rotations, future research should assess the impact of multiple away rotations on success in the Match. Allowing students limited access to letters of recommendation content would allow students and advisors to make informed decisions regarding additional rotations. Students would benefit from greater standardization of the application process for away rotations, with uniform dates, timely acceptances and rejections, and a cap for the number of applications per student. Students can maximize their returns by targeting away rotations that provide the most diverse educational experiences and valuable letters of recommendation.

Early Impact of the Western Journal of Emergency Medicine CDEM/CORD Special Issue in Educational Research & Practice.

INTRODUCTION: In 2015, with a stated goal of disseminating best teaching practices and developing a community of educational scholars, the Council of Emergency Medicine Directors (CORD) and the Clerkship Directors of Emergency Medicine (CDEM) created an annual Special Issue in Educational Research and Practice (Special Issue) in cooperation with the Western Journal of Emergency Medicine. The intention of this study was to analyze the impact of this effort to date. METHODS: Bibliometric data was gathered on all four special issues, 2015-2019, from the Web of Science and then verified with the eScholarship website. Authorship, academic affiliation, date published, article type, and format were tabulated for descriptive analysis. Using metrics from Google Scholar, alternative scholarly impact metrics (altmetrics), and the eScholarship website, the authors identified top articles and grouped them into themes. RESULTS: Of the 136 articles included in the first four years of the Special Issue, 126 represented peer-reviewed publications with an overall acceptance rate of 25.0% (126/505). Authors from this cohort represented 103 of the 182 (56.6%) Accreditation Council for Graduate Medical Education (ACGME) programs in existence at the time of the inaugural issue. Multi-institutional studies represented 34.9% (44/126) of the peer-reviewed publications. Traditional and alternative publication metrics are reported to assess the impact of articles from the Special Issues. CONCLUSION: The Special Issue is a proven outlet to share best practices, innovations, and research related to education. Additionally, the infrastructure of this process promotes the development of individual faculty and a community of teaching scholars.

Professionalism Milestones Assessments Used by Emergency Medicine Residency Programs: A Cross-sectional Survey.

INTRODUCTION: Professionalism is a vital component of quality patient care. While competency in professionalism is Accreditation Council for Graduate Medical Education (ACGME)-mandated, the methods used to evaluate professionalism are not standardized, calling into question the validity of reported measurements. We aimed to determine the type and frequency of methods used by United States (US) -based emergency medicine (EM) residencies to assess accountability (Acc) and professional values (PV), as well as how often graduating residents achieve competency in these areas. METHODS: We created a cross-sectional survey exploring assessment and perceived competency in Acc and PV, and then modified the survey for content and clarity through feedback from emergency physicians not involved in the study. The final survey was sent to the clinical competency committee (CCC) chair or program director (PD) of the 185 US-based ACGME-accredited EM residencies. We summarized results using descriptive statistics and Fisher's exact testing. RESULTS: A total of 121 programs (65.4%) completed the survey. The most frequently used methods of assessment were faculty shift evaluation (89.7%), CCC opinion (86.8%), and faculty summative evaluation (76.4%). Overall, 37% and 42% of residency programs stated that nearly all (greater than 95%) of their graduating residents achieve mastery of Acc and PV non-technical skills, respectively. Only 11.2% of respondents felt their programs were very effective at determining mastery of non-technical skills. CONCLUSION: EM residency programs relied heavily on faculty shift evaluations and summative opinions to determine resident competency in professionalism, with feedback from peers, administrators, and other staff less frequently incorporated. Few residency programs felt their current methods of evaluating professionalism were very effective.

Do Emergency Medicine Residency Graduates Feel Prepared to Manage Closed Fractures After Training?

OBJECTIVES: Fractures comprise 3% of all emergency department (ED) visits. Although emergency physicians are often responsible for managing most of the initial care of these patients, many report a lack of proficiency and comfort with these skills. The primary objective was to assess how prepared recent emergency medicine (EM) residency graduates felt managing closed fractures upon completion of residency. Secondary objectives included whether residency training or independent practice contributed most to the current level of comfort with these procedures and which fractures were most commonly reduced without orthopedic consultation. METHODS: An anonymous online survey was sent to graduates from seven EM residency programs over a 3-month period to evaluate closed fracture reduction training, practice, and comfort level. Each site primary investigator invited graduates from 2010 to 2014 to participate and followed a set schedule of reminders. RESULTS: The response rate was 287/384 (74.7%) and included 3-year (198/287, 69%) and 4-year (89/287, 31%) programs. Practice in community, academic, and hybrid ED settings was reported by 150/287 (52.3%), 64/287 (22.3%), and 73/287 (25.4%), respectively. It was indicated by 137/287 (47.7%) that they reduce closed fractures without a bedside orthopedic consultation greater than 75% of the time. The majority of graduates felt not at all prepared (35/287, 12.2%) or somewhat prepared (126/287, 43.9%) upon residency graduation. Postresidency independent practice contributed most to the current level of comfort for 156/287 (54.4%). The most common fractures requiring reduction were wrist/distal radius and/or ulna, next finger/hand, and finally, ankle/distal tibia and/or fibula. CONCLUSIONS: Although most recent graduates feel at least "somewhat" prepared to manage closed fractures in the ED, most felt that independent practice was a greater contributor to their current level of comfort than residency training. Recent graduates indicate that fracture reduction without orthopedic consultation is common in today's clinical practice. This survey identifies common fractures requiring reduction which EM residencies may wish to consider prioritizing in their emergency orthopedic curricula to better prepare their residents for independent clinical practice.

Surgical Residency Recruitment-Opportunities for Improvement.

INTRODUCTION: The Association of Program Directors in Surgery convened a panel during Surgical Education Week 2016 to discuss the current state of the general surgery residency application process and to review alternative ways to evaluate the suitability of each applicant to a residency program. METHODS/RESULTS: Over 40,000 applicants registered for the National Resident Matching Program's 2016 Main Residency Match. General Surgery had 2345 applicants for 1241 categorical postgraduate year (PGY)-1 positions, and 1239 of those positions were filled when the matching algorithm was processed. Program Directors reported that only 33% of applications received an in-depth review, and 62% were rejected with minimal review. Eventually (after all applications had been reviewed), only 13% of applicants were invited to interview. CONCLUSIONS: There are several opportunities for improvement within the current application process. These included standardized letter of recommendation and personal statements, refinement of the interview process, and recalibration of the Medical Student Performance Evaluation.

The Standardized Letter of Evaluation for Postgraduate Training: A Concept Whose Time Has Come?

A medical student's letter of recommendation for postgraduate training applications should provide a fair and accurate assessment of academic and clinical performance, as well as define character attributes pertinent to the practice of medicine. Since its inception in 1997, the emergency medicine (EM) standardized letter of evaluation (SLOE) has evolved into an instrument that provides just such an assessment. Concise, standardized, and discriminating in its assessment of performance relevant to the practice of EM, the SLOE is judged by program directors in EM as the most valuable component of a potential resident's application. Other specialties would benefit from such a specialty-specific perspective, which is currently lacking in most Electronic Residency Application Service application materials. Creation of specialty-specific SLOEs which define performance metrics or competencies and noncognitive personality traits critical to each unique specialty would add substantially to the holistic review of our graduating students. As a result, specialty-specific SLOEs would increase the likelihood that programs could effectively identify applicants who would not only be a "good fit" for their programs but also graduate to become successful physicians.

A Novel Specialty-Specific, Collaborative Faculty Development Opportunity in Education Research: Program Evaluation at Five Years.

PURPOSE: For the busy clinician-educator, accessing opportunities that develop the skills and knowledge necessary to perform education research can be problematic. The Medical Education Research Certification at Council of Emergency Medicine Residency Directors (MERC at CORD) Scholars' Program is a potential alternative. The current study evaluates the program's outcomes after five years. METHOD: The authors employed a quasi-experimental design in this study. The study population consisted of the initial five MERC at CORD cohorts (2009-2013). Development of a logic model informed Kirkpatrick-level outcomes. Data from annual pre/post surveys, an alumni survey (2014), and tracking of national presentations/peer-reviewed publications resulting from program projects served as outcome measurements. RESULTS: Over the first five years, 149 physicians participated in the program; 97 have completed six MERC workshops, and 63 have authored a national presentation and 30 a peer-reviewed publication based on program projects. Of the 79 participants responding to the pre- and postsurveys from the 2011-2013 cohorts, 65 (82%) reported significant improvement in skills and knowledge related to education research and would recommend the program. Of the 61 graduates completing the alumni survey, 58 (95%) indicated their new knowledge was instrumental beyond educational research, including promotion to new leadership positions, and 28 (47% of the 60 responding) reported initiating a subsequent multi-institutional education study. Of these, 57% (16/28) collaborated with one or more peers/mentors from their original program project. CONCLUSIONS: Kirkpatrick-level outcomes 1, 2, 3, and perhaps 4 demonstrate that the MERC at CORD program is successful in its intended purpose.

Council of Emergency Medicine Residency Directors' standardized letter of recommendation: the program director's perspective.

OBJECTIVES: The standardized letter of evaluation (SLOE) was created in 1997 to provide residency program directors (PDs) with a summative evaluation that incorporates normative grading (i.e., comparisons to peers applying to emergency medicine [EM] training). Although the standard letter of recommendation (SLOR) has become increasingly popular and important in decision-making, it has not been studied in the past 12 years. To assess the SLOR's effectiveness and limitations, the perspective of EM PDs was surveyed in this study. METHODS: After validation of the questionnaire by 10 retired PDs, the survey was sent to the PD of each of the 159 EM residencies that existed at that time. The survey was circulated via the Council of Emergency Medicine Residency Directors' (CORD) listserv from January 24, 2013, to February 13, 2013. Weekly e-mail reminders to all PDs served to increase participation. RESULTS: A total of 150 of 159 PDs (94.3%) completed the questionnaire. Nearly all respondents (149 of 150; 99.3%) agreed that the SLOR is an important evaluative tool and should continue to be used. In the application process, 91 of 150 (60.7%) programs require one or more SLORs, and an additional 55 (36.7%) recommend but do not require a SLOR to be considered for interview. When asked to identify the top three factors in deciding who should be interviewed, the SLOR was ranked first (139 of 150; 92.7%), with EM rotation grades ranked second (73 of 150; 48.7%). The factors that were most often identified as the top three that diminish the value of the SLOR in order were 1) "inflated evaluations" (121 of 146; 82.9%), 2) "inconsistency between comments and grades" (106 of 146; 72.6%), and 3) "inadequate perspective on candidate attributes in the written comments" and "inexperienced authors" (60 of 146; 41.1% each). CONCLUSIONS: The SLOR appears to be the most important tool in the EM PD's armamentarium for determining which candidates should be interviewed for residency training. Although valuable, the SLOR's potential utility is hampered by a number of factors, the most important of which is inflated evaluations. Focused changes in the SLOR template should be mindful that it appears, in general, to be successful in its intended purpose.